Supporting Efficient Noncontiguous Access in PVFS over InfiniBand

Noncontiguous I/O access is the main access pattern in many scientific applications. Noncontiguity exists both in access to files and in access to target memory regions on the client. This characteristic imposes a requirement of native noncontiguous I/O access support in cluster file systems for high performance. In this paper, we address two main issues on supporting efficient noncontiguous I/O access in cluster file systems over a high performance network. One is noncontiguous data transmission between the client and the I/O server. The second is noncontiguous disk access on the I/O server itself. We propose a novel approach, RDMA Gather/Scatter, to transfer noncontiguous data for such I/O accesses, and design a new scheme, Optimistic Group Registration, to reduce memory registration costs associated with this approach. For the second issue, we deploy data sieving on the I/O server to process a large number of small noncontiguous disk accesses. Unlike other data sieving implementations, our I/O server uses a cost model actively and intelligently to decide whether it is beneficial to perform data sieving or not. We have designed and incorporated these approaches in a version of PVFS over InfiniBand. Through a range of PVFS and MPI-IO micro-benchmarks, the MPI-IO tiled access test and the NAS BTIO benchmark, we demonstrate that our approaches attain significant performance gains compared to other existing approaches.